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Introduction and summary 


The Every Student Succeeds Act, or ESSA, renews America’s national commit¬ 
ment to public education and to the belief that all children and youth in the 
United States deserve a high-quality education regardless of their background or 
circumstances. Signed into law in December 2015, ESSA offers state education 
agencies significant opportunities to use evidence to support the improvement of 
schools and ensure better outcomes for all students. 

Much will change in practice and policy as ESSA replaces the law, regulations, 
and guidance established through the 2001 No Child Left Behind Act, or NCLB, 1 
but two elements of the new legislation stand out: the shift away from federal 
mandates toward greater state and local authority and the emphasis on evidence- 
based school improvement practices. This report addresses this second element by 
clarifying the definition of “evidence-based” that ESSA uses, distinguishing it from 
the “scientifically based research” 2 provisions of NCLB and providing a framework 
for how state education agencies can maximize collaborative efforts to implement 
evidence-based school improvement practices. 

Briefly, the evidence-based approach encourages state and district leaders to 
consider multiple tiers of evidence and examine the strength of the evidence in 
making decisions. On the other hand, scientifically based research sets a very spe¬ 
cific, narrow standard for acceptable evidence. These two terms will be examined 
in greater detail later in the report. 

Most of the early media attention given to ESSA has focused on its clear attempt 
to address the criticisms leveled at NCLB, including perceived federal overreach. 3 
Unlike NCLB, ESSA does not mandate particular school improvement activities. 
Furthermore, there is no separate so-called school improvement funding stream 
within Title I. Instead, states must now set aside 7 percent of their Title I funds—or 
the amount the state previously received under Sections 1003(a) and 1003(g) of 
NCLB, whichever is greater—to support school improvement, and they have signifi¬ 
cant flexibility in terms of how these dollars are used. 4 States may also set aside 3 per¬ 
cent of Title I funds for direct student services, such as tutoring and credit recovery. 
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These provisions offer new opportunities and new challenges for states. States must 
now plan the use of Title I funds in a way that balances an ongoing commitment 
to a statewide system of improvement for low-performing schools, including high 
schools, with other identified needs. Unlike previous versions of the law, ESSA does 
not specify the school improvement strategies that a state must pursue in support 
of district, or local education agency, efforts to improve low-performing schools. 
However, it does require states to approve and monitor local education agency plans 
for these schools to ensure that they include evidence-based approaches. 

Veteran educators and policymakers will recall that NCLB introduced the require¬ 
ment that improvement policy and practice should be grounded in “scientifically 
based research.” This standard brought both advances and frustration to the field. 

It seemed over the past decade that decision-makers were constantly hearing from 
researchers and experts that almost nothing works, while at the same time product 
and service providers were all saying that their offerings worked and were sup¬ 
ported by scientifically based research. It was hard for policymakers and educators 
to make the best choices in this environment. 

Entering the era of evidence-based school improvement, state education 
agencies are asking: Will the evidence-based policy elements of ESSA be any 
different, or will education product and service providers simply replace “sci¬ 
entifically based” with “evidence based” in their promotional materials? Will 
anything change for the better? 

The Center for American Progress and Knowledge Alliance believe that the evi¬ 
dence-based provisions of ESSA are a significant advancement in promoting the 
use of evidence to support schools in need of improvement and to achieve better 
student outcomes. CAP and Knowledge Alliance also believe that the shift away 
from federal mandates and toward greater state and local autonomy is a move in 
the right direction. Together, these changes have the potential to drive a new era of 
school improvement in the United States. 
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Authors' note: To assist state education agencies and other leaders, this report provides 
early information to clarify the definition of "evidence based" in ESSA, distinguishing 
it from the scientifically based research provisions of NCLB; discusses the promise of 
evidence-based school improvement; and suggests ways that state education agencies 
can maximize collaborative efforts to implement evidenced-based practices. 

While the term "evidence based" is used throughout ESSA, this report focuses on the 
use of this concept in relation to state and district school improvement activities. Read¬ 
ers who are interested in the latest and most authoritative ESSA guidance from the U.S. 
Department of Education should consult the department's website . 5 

This report does not attempt to provide a detailed list of conceptual and methodologi¬ 
cal issues related to the implementation of ESSA evidence tiers. Others, such as Rebecca 
Herman and her colleagues at the RAND Corporation, have raised some of these early 
questions and issues . 6 

Such issues will have to be addressed by the U.S. Department of Education and state 
education agencies as provisions in the law are translated into regulations and guid¬ 
ance. It is too early to predict what this guidance will say, but it is not too early to think 
about identifying important considerations for state education agencies as they work 
collaboratively with local districts to implement evidence-based policy and practice. 
This report identifies some of these considerations. 
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Evidence-based approaches 
as defined in the Every 
Student Succeeds Act 


To implement ESSA effectively, education leaders in state education agencies and 
other supporting organizations need a firm understanding of the new legislation 
and the specific language it uses. The law establishes three levels of evidence to 
use when choosing an approach to school improvement. These evidence levels are 
commonly referred to as Tier I, for “strong evidence”; Tier II, for “moderate evi¬ 
dence”; and Tier III, for “promising evidence.” 7 The creation of these tiers implies 
that—all things being equal—districts and schools should use improvement 
approaches with the strongest level of evidence. As discussed later in this report, 
the level of evidence is one of several important factors that state education agen¬ 
cies must help districts and schools weigh as they implement the new law. 

The law also mentions a fourth tier, based on providing what is termed a “ratio¬ 
nale” regarding the merit of an approach and requiring a commitment to “exam¬ 
ine”—that is to say, evaluate—the effects of the approach. 8 However, approaches 
in this fourth tier cannot be used for the purpose of school improvement sup¬ 
ported by Title I school improvement funds under Section 1003(a). Districts and 
schools can use strategies in this fourth tier for other school improvement efforts. 
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Excerpt from ESSA regarding evidence-based approaches 


Section 8101 (21) EVIDENCE-BASED— 

(A) IN GENERAL—Except as provided in subparagraph (B), the term "evidence-based," when used 
with respect to a State, local educational agency, or school activity, means an activity, strategy, or 
intervention that— 

(i) demonstrates a statistically significant effect on improving student outcomes or other relevant 
outcomes based on— 

(I) strong evidence from at least 1 well-designed and well-implemented experimental study; 

(II) moderate evidence from at least 1 well-designed and well-implemented quasi-experimental 
study; or 

(III) promising evidence from at least 1 well-designed and well-implemented correlational study 
with statistical controls for selection bias; or 

(ii) (I) demonstrates a rationale based on high-quality research findings or positive evaluation 
that such activity, strategy, or intervention is likely to improve student outcomes or other relevant 
outcomes; and 

(II) includes ongoing efforts to examine the effects of such activity, strategy, or intervention. 

(B) DEFINITION FOR SPECIFIC ACTIVITIES FUNDED UNDER THIS ACT—When used with respect to 
interventions or improvement activities or strategies funded under section 1003, the term "evidence- 
based" means a State, local educational agency, or school activity, strategy, or intervention that meets the 
requirements of subclause (I), (II), or (III) of subparagraph (A)(i). 

Source: Every Student Succeeds Act, Public Law 95,114th Cong. 1st sess. (December 10,2015), pp. 289-290, available at 
https://www.c 0 ngress.g 0 v/bill/l 14th-congress/senate-bill/1177/text. 
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Evidence-based vs. 
scientifically based research 


Before the advent of evidence-based approaches, the No Child Left Behind Act 
required that scientifically based research serve as the standard to determine 
which approaches could be used for school improvement. At that time, the 
concept of scientifically based research represented a significant advancement in 
public policy. It put evidence use at the center of the school improvement process. 

However, the approach offered a fairly narrow set of criteria for what evidence 
should count in school improvement decisions. In principle, it limited acceptable 
evidence to well-conducted randomized control trials, or RCTs—also known as 
experiments—and certain high-quality studies with quasi-experimental designs, 
or QEDs. The scientifically based requirement for RCTs and QEDs may prove to 
be similar to Tier I and Tier II requirements under ESSA once the Department of 
Education issues guidance. 

In practice, the scientifically based research approach created a number of dif¬ 
ficulties, including the fact that few commonly implemented school improve¬ 
ment approaches had undergone RCTs or eligible QEDs. In addition, many 
school improvement approaches that were examined through studies that met 
the scientifically based research standards were not found to have consistent 
positive effects in terms of student outcomes. Particularly, in the early years of 
scientifically based research, effective school improvement programs that met 
the requirements were in short supply. Although the past decade has seen an 
increase in the number of approaches studied using RCT and QED methodolo¬ 
gies—and more programs have been found with positive effects—the scientifi¬ 
cally based research standard proved challenging. In addition to limiting the 
types of studies that could provide evidence, it perhaps encouraged decision¬ 
makers to focus too narrowly on research design issues. 

In effect, the scientifically based research standard created a so-called thumbs- 
up/thumbs-down system, with the main question being: Will the evidence count 
or won’t it? In contrast, ESSA’s evidence-based language suggests a different 
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approach that broadens the range of acceptable evidence for improving low- 
performing schools and effectively asks: On a scale from 0 to 3, how strong is 
the evidence? This new strategy for using evidence seems more suited to the 
reality of the existing research base in education and the choices available to 
decision-makers. 

When considering what approach to take with federal school improvement 
funding, decision-makers now have the flexibility to consider any of the strate¬ 
gies that meet at least Tier III standards. This broader array of choices may help 
them find approaches that are best suited to local needs and improvement plans. 
However, with this greater flexibility comes a greater responsibility to make 
choices that reflect local needs and yield improved results. In essence, state 
and local education agencies must use their own judgment regarding available 
evidence across the three tiers. Coupled with ESSA’s greater latitude for state 
and local education agencies, the demands on states and districts to support and 
implement evidence-based school improvement have increased. 
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The benefits of an evidence-based 
approach to school improvement 


If well-implemented ; the Every Student Succeeds Act has the potential to bring 
all levels of the U.S. education system together under the banner of evidence- 
based approaches that work for children and youth—from policy leaders in 
state houses to classroom teachers. Researchers Kara Finnigan and Alan Daly 
have pointed out that the No Child Left Behind-era emphasis on scientifically 
based research has not resulted in a unified approach to ensuring that all stu¬ 
dents have effective schools. Improving education will require that policymak¬ 
ers and educators alike reject whims, fads, and promotional claims in favor of 
reliable evidence. The provisions ofESSA offer the opportunity for educators to 
implement school improvement that is informed by a shared understanding of 
evidence and its appropriate application. Over time, particularly if coupled with 
a robust evaluation effort, evidence-based school reform offers the opportunity 
for continuous improvement in education. 

Using evidence to meet the needs of all students—particularly those who have 
been traditionally underserved—is both a moral imperative and a practical one. 
Struggling schools should not be left to their own devices to improve by trial 
and error—not when there are approaches that have a legitimate evidence base 
and not when time and resources are at a premium. School improvement expert 
Rebecca Herman and her colleagues make this point when they write: 

Investments in education must produce results. Students' efforts, teachers' time, 
and scarce financial resources are more likely to be well spent when education- 
improvement activities are selected because there is evidence that they are effec¬ 
tive. To select education-improvement activities without considering their prior, 
proven impact may be seen as an irresponsible use of limited resources. 
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CAP report provides an example of how evidence can 
support school improvement 


In March 2016, CAP issued a report that identified strategies to improve low-performing 
schools under ESSA. 12 The report—"Strategies to Improve Low-Performing Schools 
Under the Every Student Succeeds Act: How 3 Districts Found Success Using Evidence- 
Based Practices"—examined the success of schools in three districts implementing a 
set of five policies that researchers Will Dobbie and Roland Fryer identified: frequent 
teacher feedback; data-driven instruction; high-dosage tutoring; increased instruc¬ 
tional time; and a relentless focus on academic achievement. The researchers found 
that this set of policies "explains almost half of the variation in school effectiveness" in 
charter schools they studied. 13 The authors note, "While there are important caveats to 
the conclusion that these five policies can explain significant variation in school effec¬ 
tiveness, our results suggest a model of schooling that may have general application." 14 

The CAP report focused on a subset of seven improving public schools across three 
districts that used the five policies identified by Dobbie and Fryer in their high-per¬ 
forming charter school study. Through interviews with key stakeholders in each district, 
the authors tried to identify what it takes to implement these policies effectively. The 
strategies identified were: 

• Allot a sufficient amount of planning time 

' Implement school-level budgeting and strategically reallocate funds based on 
student needs 

• Aggressively recruit, hire, and train innovative school leaders and teachers 

• Obtain stakeholder investment via data and word-of-mouth 

This suggestive study may help district and school leaders use these evidence-based 
practices successfully to improve outcomes in their own schools. 
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Recommendations 


Implementation of the evidence-based provisions of the Every Student Succeeds 
Act will be a complex and challenging undertaking. If done right, effective policy 
implementation will help yield significant and sustained school improvement and 
better outcomes for students. 

Many of the implementation considerations will be technical in nature, such as 
providing greater clarity regarding the types of evidence that do or do not meet 
Tier I, Tier II, or Tier III standards; determining whether every aspect of a school 
improvement approach or only a subset of elements need to be evidence based; 
and deciding which outcomes are eligible for review in determining whether an 
approach is evidence based. These questions will be answered in due course and 
are beyond the scope of this report. However, suggestions about collaborative 
implementation may be timely and can help state education agencies work in part¬ 
nership with local education agencies and the Department of Education to begin 
implementing evidence-based school improvement strategies. 

The following eight recommendations are based on a March 2016 focus group 
of former state and local education agency employees; informal interviews with 
education administrators and improvement experts, also conducted in March 
2016; and the reflections of the authors based on their school improvement work 
over the past two decades. The recommendations encourage state education agen¬ 
cies to work with national and local governmental and community agencies to act 
as a single, cohesive system that promotes clarity and coordination to achieve the 
promise of evidence-based school improvement. 


1. Define roles in the policy implementation 
process and create a theory of change 

State education agencies should start work on a theory of change that clearly speci¬ 
fies the roles of leaders at each level in the system and the steps each must take to 
make this policy initiative work. While each institutional level—national, state, 
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and local—should clearly define what role it will play in the process, state educa¬ 
tion agencies are positioned at the center of the system and are best equipped to 
articulate connections between their role and the federal role, as well as between 
their role and that of the local education agency. Many state education agency lead¬ 
ers already have experience creating theories of change and may have created these 
as part of their statewide systems of support prior to ESSA. Others maybe new to 
creating theories of change. For those new to this work, documents from The Aspen 
Institute, Organizational Research Services, and ActRnowledge may prove useful. 


Although the implementation of evidence-based ESSA provisions is new, many state 
education agency administrators have prior experience in rolling out complex initia¬ 
tives, including those that use evidence to drive the improvement process. To foster 
coordinated and effective action, veteran administrators at the state level should 
mobilize to help identify and apply the positive and cautionary lessons learned in 
the implementation of prior federal programs and initiatives such as Comprehensive 
School Reform, Reading First, supplemental educational services, the Investing in 
Innovation Fund, Race to the Top, and School Improvement Grants. The challenges 
and successes that each of these initiatives experienced—for example, in establishing 
partnerships with external service providers or in identifying who would serve as the 
authoritative source to answer questions as they emerged—can be applied to this 
policy implementation challenge. 


2. Support the use of evidence clearinghouses 
to identify promising approaches, rather than 
creating new state-approved lists 

The goal of ESSA provisions should be to encourage local education agencies to 
make judicious choices among evidence-based options and then to implement these 
approaches well. ESSA does not ask state education agencies to produce lists of 
approved programs. The work of reviewing evidence-based approaches has already 
been done by a number of clearinghouses. Providing state education agency-selected 
lists of approved or preferred approaches may serve to narrow local education 
agency choices prematurely or to promote a compliance mentality in which making 
a good match receives insufficient attention and the goal becomes merely to check 
the box that indicates that the selection process has been completed. Rather, state 
education agencies should focus on developing clear guidance and useful guides, 
pointing to existing evidence clearinghouses, and offering supports to encourage a 
reflective decision-making process by local education agencies. 
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While state education agencies can promote positive decision-making among 
local education agencies, they are not in the position to define the terms in 
ESSA—such as the four tiers of evidence—or to issue official guidance about 
ESSA. Instead, the Department of Education is best suited to address many of 
the highly technical questions that have been raised regarding evidence-based 
elements of the law, which require a consistent answer across all states. Examples 
of these concepts can be found in the previously mentioned RAND Corporation 
report and include: What types of evidence qualify for Tiers III and IV? Can 
rigorous studies with some flaws be included? Can a body of evidence, rather than 
individual findings, be considered? 

Even after these questions are answered, however, state education agencies will 
need to direct local education agencies to guidance and support. Fortunately, state 
and local education agency decision-makers are not left on their own to look for 
evidence-based approaches. In the nearly 15 years since the Institute of Education 
Sciences created the What Works Clearinghouse, the amount of evidence regard¬ 
ing the effectiveness of education approaches has grown significantly. Sources that 
offer evidence reviews that educators can consult include: 

• What Works Clearinghouse. This is a first stop for many education decision¬ 
makers seeking evidence of program effectiveness. Funded by the Department 
of Educations Institute of Education Sciences, the What Works Clearinghouse 
website offers more than 700 publications and catalogs and more than 11,000 
reviewed studies in its database. 

• Results First Clearinghouse Database. This website was created by Results First, 
a project funded by The Pew Charitable Trusts and the John D. and Catherine T. 
MacArthur Foundation to identify evidence-based programs in education and 
other related fields. It provides links to eight clearinghouses that review a total of 
nearly 1,300 programs for their effectiveness. 

• Best Evidence Encyclopedia. Sponsored by Johns Hopkins University School 
of Educations Center for Data-Driven Reform in Education, this “encyclope¬ 
dia” offers access to research syntheses and program reviews of more than 900 
programs and approaches in the areas of math, reading, science, early childhood, 
and whole-school improvement. 
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These sources for reviews of evidence-based approaches allow users to search by 
topic areas such as literacy math, and science (in Best Evidence Encyclopedia); 
special populations (in Results First Clearinghouse Database); and English lan¬ 
guage learners (in What Works Clearinghouse). These searches produce a list of 
approaches tailored to each topic area and provide reviews for considering the 
quality of these approaches. Using this type of search to identify evidence-based 
approaches engages local education agencies actively in the decision-making process. 


3. Encourage robust decision-making processes 

Making a good match between available options and school or district needs is 
perhaps the most critical element in achieving good results from an evidence- 
based improvement approach. Writing nearly two decades ago, Sam Stringfield, 
school improvement researcher, pointed out that, “The key is to approach the 
choice of a school reform [approach] as an important and complicated consumer 
decision.” Therefore, it is not enough for schools or districts to ask: Which of the 
school improvement approaches we are considering has the strongest evidence? 
By itself, this question is much too simplistic and may make it more difficult to 
achieve the district’s desired outcomes. Rather, districts and schools should con¬ 
sider asking: Of the evidence-based approaches available to us, which one is most 
likely to meet our needs and align with our improvement plan? 

Indeed, there are many factors beyond research evidence that influence a school 
improvement decision, (see Figure l) Every choice in school improvement is 
likely to be influenced by multiple factors, including the preferences and values of 
the stakeholders, such as school board members, parents, and community mem¬ 
bers; organizational contexts, such as the size and capacity of a school district, reg¬ 
ulations, and labor agreements; the professional experiences of district educators 
and administrators; and a host of other factors, including political considerations. 

The goal when making a choice should be to weigh appropriately the evidence 
for various approaches within the local context in which the decision is made, 
based on a robust needs assessment and with consultation from stakeholders and 
practitioners. If multiple evidence-based approaches are an excellent fit, then the 
comparative strength of the evidence could determine the choice. But to focus too 
narrowly on a decision based only on the strength of evidence is to court failure in 
the acceptance and implementation of even the best program or practice. Instead, 
state education agencies should urge leaders to convene decision-makers and 
address each of the four circles in Figure 1 by answering the related questions, 
weighing the input, and finding an evidence-based intervention that fits best. 
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FIGURE 1 

Factors that influence decision-making in education 


Which approaches have 
Tier I, II, or III evidence 
supporting their use? 

I 


Which approaches 
Stakeholder do stakeholders, 

preferences — such as parents and 
and values business leaders, 

prefer and value? 

organizational actors, 
and circumstance 

T 

Which approaches best fit the 
local context based on a robust 
needs assessment? 


Research 
and evaluation 
evidence 

Which approaches 

are viewed favorably Practitioner 

by teachers, y experience 

principals, and other and judgments 

expert practitioners? 

Context, 



Source: Adapted by authors from Rob Briner, David Denyer, and Denise Rousseau, "Evidence-Based Management: Concept Cleanup Time?", 
Academy of Management Perspectives 23 (4) (2009): 19-32. 


4. Help ensure high-quality services from improvement providers 

Services from curriculum and intervention providers can vary over time. For 
example, participation in a rigorous research study could raise the stakes for a 
provider to deliver results, while a later rollout might not exert the same pressure. 
Given the potential for fluctuation, below are a few things that state and local edu¬ 
cation decision-makers should keep in mind as they work with potential providers 
of evidence-based school improvement strategies: 

• Make sure the provider addresses local needs. Throughout the process continu¬ 
ally ask: How will the curriculum and/or intervention serve our school, district, 
and state goals? How will the provider make sure everything works well in our 
context? Make sure that the evidence-based approach chosen is well-aligned to 
the improvement goals that have been developed based on a data-driven planning 
process. Before choosing any school improvement approach, ask the provider to 
be clear about what actions need to occur at the state, district, or school level in 
order to achieve desired outcomes. The connection between the approach being 
considered and the desired outcomes must be explicit, coherent, and clear. 
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• Exercise consumer power. As decision-makers, state, district, and school lead¬ 
ers can demand to know more about an approach than is publicly advertised. 
Remind leaders that they should ask as many questions as necessary to be con¬ 
fident that they are choosing the right provider for their schools. This includes 
asking for references, recent financial audits, and management and staff capac¬ 
ity. Decision-makers should treat the inability or unwillingness of a provider to 
answer in-depth questions as a red flag. 

• Be appropriately skeptical. Even proponents of the most well-meaning evidence- 
based approaches will have a tendency to provide the so-called best case picture of 
their evidence and how the approach operates. Decision-makers must make sure 
the evidence is convincing. And after adoption, all those involved in implementa¬ 
tion should be open to adjusting their actions as the effort rolls out. See recom¬ 
mendation 8 below for additional suggestions related to continuous improvement. 

In many states, state education agencies will make decisions about choosing pro¬ 
viders and will therefore be the entity to ask questions and demand high-quality 
services. In other states, districts will take the lead on making these choices. When 
districts lead, state education agencies have a strong role in empowering districts 
to ask for the best possible services. 


5. Promote and provide frequent, accurate, 
and timely communication 

Even the most well-planned educational improvement initiatives can founder under 
the weight of inaccurate, inadequate, or incoherent communication or through 
the lack of appropriate engagement of the stakeholders who will be involved in the 
reforms. It is a long communications-and-action chain from the federal level to 
individual schools where the changes actually take place. 9 State education agencies 
are at the center of this chain of communication and, therefore, have an essential role 
in ensuring high-quality and timely information sharing. 

In particular, state education agencies should engage local education agencies 
early and often in the design of any statewide evidence-based school improvement 
initiative. The importance of this effort is highlighted by ongoing research on how 
individuals at the school and district levels understand the implementation of sev¬ 
eral state-level education policies. Researcher Jal Mehta recently wrote about his 
joint research with Renee Rinehart, saying, “Not surprisingly, from the teachers in 
particular, we have found absolute puzzlement and/or outright hostility about the 
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ideas and even motives of those further up the chain.” 10 To improve implementa¬ 
tion, they urge state education agencies to involve “the various stakeholders who 
would be affected by a given policy with really careful attention paid to the voices 
of those who would be most affected by it.” 11 

Three elements of communication are paramount for state education agencies early 
in the implementation process. First, states should communicate about the impor¬ 
tance of using evidence for improvement. Second, states should engage stakeholders 
early and often to gather feedback, promote knowledge, and increase buy-in regard¬ 
ing the implementation of these provisions. Finally, states should make sure commu¬ 
nication is a two-way process so that the state education agency can learn and adjust 
its policies based on the realities that districts and schools encounter. 


6. Partner with intermediaries to promote 
and support effective implementation 

State education agencies should make sure that they mobilize their systems and 
networks of intermediary organizations—such as technical assistance initiatives 
they sponsor and professional associations with which they interact—to support 
the implementation of the evidence-based provisions of the law. Intermediary 
organizations play a critical role in providing educators and policymakers with 
information and support that significantly influence their actions. 

State education agencies should actively request assistance from the Regional 
Educational Laboratories and the Comprehensive Centers funded by the U.S. 
Department of Education, and these entities should proactively provide guidance 
and technical assistance suggestions for state and local education agencies beyond 
those that specifically request support. ESSA already provides some guidance, 
specifying that the Regional Educational Laboratories “shall provide technical 
assistance” upon request to any “state or local educational agency in meeting the 
requirements of section 8101(21)”—that is, the section specifying the evidence- 
based provisions of the law. 

In addition, state education agencies should reach out to their education service 
agencies, which exist in 45 states and reach more than 80 percent of public school 
districts, 12 to ensure that these agencies are knowledgeable about these provisions and 
effectively address or refer requests for assistance to the appropriate service providers. 
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State education agencies should also consider conducting extensive and ongoing 
outreach to national professional associations and organizations that represent 
critical stakeholder groups such as administrators, teachers, state agency leaders and 
staff, school boards, state policymakers, and education-focused policy organizations. 
The importance of connecting to these groups is highlighted by the findings of a 
recent study from the National Center for Research in Policy and Practice, which 
surveyed a nationally representative sample of district and school leaders working 
in large and medium-sized local education agencies regarding their use of research. 
The survey explored “how these educational leaders use research to inform their 
decision-making,” as well as “leaders’ attitudes toward research, their efforts to 
acquire it, and the culture of research use in their organizations.” 13 One of the most 
significant findings is that of the 14 identified sources of research that administrators 
use, respondents reported that they “were most likely to access research through 
professional associations and professional conferences.” 14 


7. Facilitate effective implementation in districts, 
schools, and classrooms 

Implementation has a profound effect on program impact. Choosing an evidence- 
based program is like making any complicated consumer decision—that is to say, 
what you do after you buy a product and open the box makes a difference too. If, 
for example, there are instructions about assembling and using your purchase, 
it’s generally a good idea to read and follow those instructions. State education 
agencies need to help local education agencies build effective implementation 
plans that get down to the school and classroom levels. This can be part of existing 
school improvement planning. 

A recent evaluation conducted by Education Northwest illustrates this point. A 
large school district implemented a new reading program that had evidence of 
effectiveness. Despite the program’s past success, after a year of implementation, 
the evaluation data showed that students in the comparison classrooms did better 
on reading tests than those in the program classrooms, and these differences were 
statistically significant in one of the three grades examined. 15 So what went wrong? 

Observations showed that only about half of the teachers implemented the program 
as intended. Further investigation in the grade with statistically significant negative 
results showed that in classrooms with low or medium levels of implementation, 
students had significantly lower achievement. Meanwhile, in classrooms with high 
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levels of implementation; student achievement did not differ significantly from that 
of students participating in the regular curriculum. End-of-year survey data showed 
that many teachers believed the program was ineffective, and interview data showed 
that several key administrators and teachers questioned whether the program was a 
good fit for the district. In short, implementation failed. 16 

For this particular district, the implementation failure was fatal. After rolling out 
the program with poor implementation, it seemed extraordinarily difficult to 
retrain the teachers and get their full participation. It was just too hard to reassem¬ 
ble the product and restart it, using it as intended. Leaders wisely chose to start 
over with a different evidence-based reading program. 17 

State education agencies need to require implementation planning as part of 
district and school improvement plans and support these plans with technical 
assistance as needed. The text box below provides more information about what 
studies conducted by Education Northwest have shown regarding the ways dis¬ 
tricts and schools can ensure effective implementation of improvement initiatives 
right down to the classroom level. 


Effective local implementation: The engine of school 
improvement 

Materials and training are important components of evidence-based programs, but 
local implementation drives the program. A recent Education Northwest evaluation of 
implementation of an evidence-based program showed a surprising negative impact 
on student achievement. Closer examination through classroom observations, teacher 
surveys, and interviews showed that implementation was the likely problem. In fact, 
school staff members had such a negative view of the program that district leaders 
wisely decided to end it and choose a new evidence-based program. 

In light of the discontinuance of the program—and to guide future actions—the 
district and its funders asked Education Northwest to provide them with a new study 
identifying the factors that contribute and detract from implementation success. The 
report included a review of the research literature on program selection and imple¬ 
mentation and in-depth interviews with representative district stakeholders. The two 
sources of information suggested similar success factors, which can act as a guide for 
anyone pursuing evidence-based school improvement. 
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The literature review on program selection and program implementation suggested 

that the following elements contribute to program success: 

• Obtain strong buy-in from all involved in the initiative and create a sense of shared 
leadership, including strong community support and engagement. 

• Schedule adequate planning time, including the collection of baseline data to better 
target the initiative, and create stable planning committees. 

• Ensure adequate resources (staff, funding, time) and provide capacity building as 
needed. 

• Provide strong leadership with internal oversight and accountability. 

• Maintain clear communication with all involved. 

• Remain adaptable as the initiative progresses. 

• Provide a long-term commitment to adequate funding for sustainability. 

• Obtain an external evaluator for the initiative and monitor and evaluate long-term 
initiatives at interim times during implementation, not just at the end. 

In addition to these factors, district stakeholders pointed to three issues that states and 

districts should consider in the selection process: 

' Follow established district processes. Many districts already have processes in place 
to adopt new programs and practices. It is important to ensure that the selection of 
an evidence-based school improvement process is aligned to these processes or, if a 
different selection process is used, to create conditions that make it clear why such a 
decision was made. 

• Review existing evidence of the new program's effectiveness. States and districts 
should understand that the review process must be seen as objective and rigorous. 

• Ensure that the new program aligns with current district policies, goals, assessments, 
and needs. This is a reminder that the degree to which an evidence-based improve¬ 
ment approach fits with the identified needs, processes, and culture within a district is 
a critical consideration in selecting an approach. 


Source: Caitlin Scott, Jason Greenberg Motamedi, and Angela Roccograndi, "Making Lemonade: Learning From Program Selection 
and Implementation Challenges," Paper presented at the annual meeting of the American Education Research Association, San 
Francisco, California, April 27,2013. Available on request. 
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8. Promote continuous improvement and collective learning 


States have new opportunities through ESSA for innovation and application of 
evidence to foster sustained and sustainable school improvement. These oppor¬ 
tunities will grow out of federal policies and will, in turn, influence local school 
improvement efforts. State education agencies will play a central role in helping 
leaders engaged in evidence-based school improvement at every level take an 
approach that integrates continuous improvement. 

Improving schools requires addressing many complex challenges at once. Given 
these challenges, even the most well-designed plan that is based on the best evi¬ 
dence-based approaches will take time to succeed. Evidence regarding this factor 
was provided in a 2006 report on a national, multiyear study of the Comprehensive 
School Reform Program. Researcher Daniel Aladjem and his colleagues concluded 
that the program “works when given enough time. We observed no effects until 
years 3-5 of implementation.” 18 Based on this observation, once an evidence-based 
school improvement plan has been designed and implemented, decision-makers 
should monitor progress but also give the plan time to succeed. 

State education agencies also should provide local education agencies with addi¬ 
tional supports so that leaders can use data effectively for school improvement. 
Midcourse corrections will be necessary as any approach encounters the reality of 
the varied needs and contexts of schools in need of improvement. Ongoing use of 
data is critical for making these corrections. 

In addition, evaluation plays a critical role in the effective implementation of 
evidence-based school improvement initiatives. State education agencies must 
build an evaluation component into every aspect of policy implementation. 
While informal data use can support many of the needed course corrections, 
only an independent, ongoing evaluation specifically designed to measure both 
implementation fidelity and program impacts can address the many issues of 
practice and policy that need to be answered systematically, such as: Does our 
approach seem to be working, for whom, and under what conditions? Should 
we make changes to improve results? In the long term, summative evaluations 
can also guide large-scale revisions to policy and practice. Without evaluations, 
billions of taxpayer dollars could be wasted because of failures to adjust in real 
time or to learn from successes and failures. 


20 Center for American Progress | Better Evidence, Better Choices, Better Schools 




Finally, state education agencies can play a critical role in supporting collective learn¬ 
ing about the effective use of evidence for improvement purposes. State education 
agencies should consider convening periodic large-scale meetings to share experi¬ 
ences and provide input for program improvement. In addition, they should con¬ 
sider networks or learning communities that can help address the needs of particular 
stakeholders. The federal Comprehensive Centers have proven to be strong partners 
to state education agencies in designing and supporting a variety of professional 
development and learning communities and can serve as partners to states as they 
promote collective learning about effective evidence-based school improvement. 
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Conclusion 


There are no foolproof evidence-based school improvement approaches. 
Therefore; there are no perfect choices. Every decision a state, district, or school 
makes to improve student outcomes will be a trade-off among many priorities, 
options, and potential outcomes. Choices supported and guided by effective 
decision-making processes and strong implementation, however, will be better 
choices. State education agencies can start work now to ensure better choices that 
help create better schools. 

It cannot be known with absolute certainty what results will be achieved when 
school leaders select evidence-based school improvement approaches because no 
one can predict the future with 100 percent accuracy. What is known, however, is 
that evidence-based approaches—when wisely chosen and implemented to meet 
specific local needs—offer the best chance of success. This report and the infor¬ 
mation presented within it will hopefully help state educators and policymakers 
seize the opportunity afforded by the Every Student Succeeds Act and meet the 
promise of evidence-based school improvement. 
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